User interface for text input and virtual keyboard manipulation

ABSTRACT

A user interface allows a user to input text, numbers and symbols in to an electronic device through a touch sensitive input and perform control functions with one or more swipes gestures, swipe and hold gestures and multiple finger tap gestures. The system can differentiate the swipes and perform functions corresponding to the detected swipes based upon swipe direction, number of fingers used in the swipe and the context of the text input.

CROSS REFERENCE TO RELATED APPLICATIONS

This application claims priority to U.S. Provisional Patent ApplicationNo. 61/905,820, “User Interface For Text Input and Virtual KeyboardManipulation” filed Nov. 18, 2013, and U.S. Provisional PatentApplication No. 61/895,352, “User Interface for Text Input and VirtualKeyboard Manipulation” filed Oct. 24, 2013, both of which are herebyincorporated by reference in their entirety.

FIELD OF INVENTION

This invention relates to user interfaces and in particular to text,number and symbol input and correction on touch screen input devicessuch as smart phones and smart watches.

BACKGROUND OF THE INVENTION

The present invention relates to devices capable of recording fingermovements. Such devices include, for example, computers and phonesfeaturing touch screens, or other recording devices able to record themovement of fingers on a plane or in three-dimensional spaces.

A number of devices where finger interaction is central to their usehave recently been introduced. They include mobile telephones (such asthe Apple iPhone, the Samsung Galaxy S), tablet computers (such as theApple iPad, or the Blackberry Playbook), as well as a range of mobilecomputers, tablets, smart watches, PDAs and satellite navigationassistants. The growth in the use of smartphones, tablets and smartwatches in particular has accelerated the introduction of touch screeninput for many users and uses.

In some devices featuring a touch screen, it is common for systems toemulate a keyboard text entry system. The devices typically display avirtual keyboard on screen, with users tapping on the different lettersto input text. The lack of tactile feedback in this typing process meansthat users are typically more error prone than when typing on hardwarekeyboards.

Most text correction systems feature a combination of auto-correctingand manual-correcting (or disambiguation) functionality. Typically, thesystem will attempt to guess and automatically correct common typingerrors. However, many systems perform the auto-correction without anyindication of the corrections. Thus, the user must constantly watch whatthe system is inputting and make manual corrections if anauto-correction error is detected which can slow the text input process.Other correction systems give the user the ability to reject anautomatic correction, or manually select an alternative one.

A common problem with such systems is that the user is required to beprecise in their typing, and also to be precise in their operation ofthe auto- and manual-correcting functionality. Such operation typicallyrequires the user to interact with the touch screen by pressing onspecific areas of the screen to invoke, accept, reject, or changecorrections.

The present invention describes a suite of functions allowing users amuch more intuitive, faster and accurate interaction with such a typingsystem. The resulting system is dramatically more accessible and easy touse for people with impaired vision, compared to other existing systems.

SUMMARY OF THE INVENTION

The present invention is directed towards a system that interpretsgestures input thorough a touch screen of an electronic computer devicesuch as touch screen computers, tablets, smart watches, etc. Theelectronic devices can operate software that allows text to be input.The text can be input through a virtual keyboard on the touch screen orthrough any other text input systems such as voice to text mechanisms.

The system can input text by tapping on the virtual keyboard on thetouch screen. The system can also detect the type of gesture, thedirection of the gesture and the context of the text that has beeninput. Based upon the type, direction and context of the gesture, apredetermined function can be performed by the system. The functions canbe grouped according into several types of gestures. For example, singlefinger swipe gestures can perform various tasks based upon the directionand context of the text input. A single finger right swipe in thecontext of a partial word can cause the system to replace the partialword with a first suggested word and insert a space after the firstsuggested word. If the first suggested word is not the intended word,the user can input a one finger up/down swipe gesture(s) to replace thefirst suggested word with a second suggested word. The user can repeatthe down swipe gestures to scroll through the suggested words until thesuggested word matches the desired word. The user can perform up swipesto return to the previous suggested words and the system will displaythe original input text after the first suggested word. Once the desiredword is input the user can resume tapping on letters to input a nextword.

Once the user has input the last word of a sentence, the user can inputa one finger or a two finger right swipe. The system can interpret thecontext of a one finger or two finger right swipe immediately after aword has been input as a user request for a punctuation mark. The normalpunctuation mark input can be a period. However, if this is not thedesired punctuation mark, the user can input single finger up or downswipes as described above to input other suggested punctuation marks. Inother embodiments, the inventive system can analyse the context of theinput text and initially suggest a punctuation mark such as a questionmark or exclamation mark in response to two sequential right swipes. Insome embodiments, the system can analyse the types of words that havebeen input in the sentence. If the first word of the sentence is aninterrogatory type word such as Which, How, What, Why Where, When, Will,Are, Can, Is, etc. can indicate that the sentence is a question. In anembodiment, the inventive system can detect the presence ofinterrogative words and may input a question mark “?” rather than aperiod “.” in response to a punctuation mark gesture.

In an embodiment, the system can predict the next word that the userwould like to input and may display a set of suggested next words. Thisnext word prediction can be based upon an n-gram methodology wherein alarge volume of writings can be analysed to determine probabilities ofdifferent multiple word combinations. The predicted words can bedisplayed in a suggested word area and can be arranged based upon thestatistical probability of the word combinations. In an embodiment, theuser can use a single finger right swipe to input the first predictedword. In an embodiment, if the first predicted word is not the desiredword, the user can either use up/down swipes to scroll through thepredicted word list or input individual text letters to manually inputthe next word. In some embodiments, the suggested words can include bothwords and punctuation marks.

The system can also interpret right swipe and hold gestures as requeststo repeat the last input. For example, if the last input was a periodpunctuation mark a subsequent right swipe and hold will result inadditional periods being input. The input can be repeated until theswipe and hold gesture is released.

The system can interpret left swipes is delete functions depending uponthe context of the text input. For example, a left swipe while in themiddle of a word can delete a single letter. However, if the left swipeis after an entire word is input, the system can delete the entire word.A left swipe and hold can repeat the prior deletion functions until thesingle finger left swipe and hold gesture is released.

In addition to text controls, the inventive system can detect gesturesfor performing other non-text input of functions. For example, the usercan input additional words to the system dictionary using gestures. Whena sequence of characters is input, the system will include the exacttext input and the suggested words in the sequence: exact text input,first suggested word, second suggested word, third suggested word, etc.So if the system responds to a right swipe by changing the exact inputtext with the first suggested word, the user can up swipe and theoriginal exact input text can be displayed and input. A subsequentsingle finger up swipe can cause the exact text to be input to thedictionary. If the user does not want to add the word to the dictionaryan immediate single finger down swipe can cause the exact text to beremoved from the dictionary.

The system can also interpret gestures to change keyboards and keyboardlayouts. The system can include various keyboards including a basicQWERTY keyboard as well as other number, symbol and foreign languagekeyboards. In an embodiment, the system can include a keyboard key(s)which can be tapped to change the keyboards being used by the device. Inmany devices, the touch screen can be very small and it must provide anarea for text input and an area for input text display. The inventivesystem can allow a user to make the keyboard text input sectioninvisible to make more of the screen available for displaying inputtext. In an embodiment, the inventive system can interpret two finger upor down swipes as controls for changing the displayed keyboards, whichcan be switched between a full keyboard, a minimal keyboard and aninvisible keyboard. Because many users can memorized the layout of aQWERTY keyboard, it may not be necessary to display this basic keyboard.

Some devices such as smart phones are held in one hand and text is inputwith the user's thumb which can be on the left or right side of thetouch screen depending upon which hand is holding the phone. In anembodiment, the inventive system can respond to a two or a three fingerleft swipe gesture by shifting the keyboard to the left of the touchscreen which can be more comfortable for left handed users. Conversely,the system can interpret a two or three finger right swipe as a commandto shift the keyboard to the right side of the touch screen.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates an embodiment of a device that includes a virtualkeyboard displayed on a touch screen and hardware buttons;

FIG. 2 illustrates a block diagram of system components;

FIGS. 3-12 illustrate embodiments of the device performing text inputand word correction gesture functions;

FIGS. 13A and 13B illustrate embodiments of the device performingdictionary control gesture functions;

FIGS. 14-17 illustrate embodiments of the device performing punctuationgesture functions;

FIGS. 18-21 illustrate letter deletion and word deletion gesturefunctions;

FIGS. 22-28 illustrate embodiments of text input, word correction, wordprediction and punctuation mark input gestures;

FIGS. 29-34 illustrate letter keyboard and displayed layout gesturefunctions;

FIG. 35 illustrates a right side keyboard compression gesture;

FIG. 36 illustrates a left side keyboard compression gesture;

FIGS. 37-38 illustrate smart watch embodiments of a device that includesa virtual keyboard; and

FIG. 39 illustrates an embodiment of cameras used to detect gesturesused with embodiments of the inventive system.

DETAILED DESCRIPTION

The invention is directed towards gesture controls input to a devicecomprising a display capable of presenting a virtual keyboard which canbe any area where the user input text can be displayed and atouch-sensitive controller such as a touch pad or a touch screen.However, in other embodiments, a screen or a touch-sensitive controllermay not be required to perform the method of the claimed invention. Forexample, in an embodiment, the input device can simply be the user'sbody or hands and a controller that is able to understand the user'sfinger movements in order to produce the desired output. In addition tothe finger inputs, in other embodiments such as tablet and smart watchapplications, other device sensors can be used as input devices. Forexample, internal device accelerometers and gyroscopes can be used inconjunction with the touch screen input to control the operation andinput text to the device. The output can be either on a screen orthrough audio signals. For example, the input device may be a camerasuch as a Microsoft Kinect controller that is directed at the user. Thecameras can detect the movement of the user and the output can betransmitted through speakers or other audio devices such as headphones.Optionally, the output can be transmitted through an output channelcapable of audio playback, such as speakers, headphones, or a hands-freeear piece.

In some embodiments, the device may be a mobile telephone, a tabletcomputer or a smart watch. In such cases, the text display andtouch-sensitive controller may both be incorporated in a singletouch-screen surface or be separate components. With the inventivesystem, the user controls the electronic device using thetouch-sensitive controller in combination with performing a number of“gestures” which are detected by the touch-sensitive controller. Someexisting systems are capable of detecting gestures input to atouch-sensitive controller such as U.S. Patent Publication Nos. US2012/0011462, US 2013/0021248 and US 2013/0212515, which are each herebyincorporated by reference.

The inventive system may be programmed to recognize certain types ofgestures including: single finger and multiple finger tapping, swipingand combinations of taps and swipes. Tapping at different areas of thescreen and different quantities of taps. For example, the system candistinguish between a single tap, a double tap, a triple tap, aquadruple tap, etc. The multiple taps can be by the same finger ormultiple fingers such as two finger taps, three finger taps, four fingertaps, etc. In yet another embodiment, the system can detect multipletaps with different fingers. For example, a first tap with a firstfinger, a second tap with a second finger, a third tap with a thirdfinger and a fourth tap with a fourth finger. These multiple taps, canalso include any variation or sequence of finger taps. For example, afirst tap with a first finger, a second tap with a second finger, athird tap with a first finger and a fourth tap with a third finger. Thedisclosed tapping can be described as “tap gestures.”

Swiping can include touching the screen and sliding the finger acrossthe screen in different directions across the screen and a differentlocations on the screen. Swiping can also be performed using one or morefingers. The system can differentiate these different swipes based uponthe number of fingers detected on the screen. The system may be able todistinguish between linear swipes and rotational swipes. Linear swipescan be detected as a touching of the input at a point a movement whilemaintaining contact in a specific direction which can be up, down, left,right and possibly diagonal directions as well such as: up/right,up/left, down/right and down/left. Rotational swipes can be detected asa touching of the input at a point and a circular movement whilemaintaining contact. The system can detect clockwise andcounter-clockwise rotational swipes.

The system may also detect combinations of gestures. For example, alinear swiping gesture as described above followed by holding the fingeron a screen for a short time before releasing. The holding of the fingeron the screen can be described as a “hold gesture” and the combinationof the swipe and hold can be described as a “swipe and hold” gesture.This combination gesture can include holding the finger(s) against thescreen at the end of the swipe for a predetermined period of time whichcan be longer than 1 second.

Typically, the user will use tap gestures to type the individual lettersused to create words on a virtual keyboard, emulating a typing movement.Unlike most virtual keyboards, there may not be any control keys such asspace, backspace and shift. Instead these functions can be performedusing other touch gestures. In an embodiment, all tapping, swipes andother detected gestures must take place within the designated keyboardarea of the touch input device which can be the lower part of a touchscreen where a virtual keyboard and editing information may bedisplayed.

The inventive system can also correct the user's text input as he types,using an algorithm to identify and analyse typing errors. When thesystem detects the user may have made such an error, the correctionalgorithm will provide alternative suggestions on an optical display orvia audio feedback.

The user can navigate through the correction algorithm suggestions usinga set of defined swipe and swipe-and-hold gestures. Additionally, theuser may be able to insert symbol characters, and to format the text,using either swipe and/or swipe and hold gestures. Typically, allgestures will be restricted to some area of the touch surface, mostcommonly the area of the onscreen keyboard. However, in an embodiment,the inventive text input system can detect gestures on any portion ofthe touch screen input device. The present invention will thus provide acomprehensive text input system incorporating spelling/typing check,format, and advanced input, by detecting applicable gestures.

With reference to FIG. 1, a top view of an exemplary electronic device100 is illustrated that implements a touch screen-based virtual keyboard105. The illustrated electronic device 100 includes an input/display 103that also incorporates a touch screen. The input/display 103 can beconfigured to display a graphical user interface (GUI). The GUI mayinclude graphical and textual elements representing the information andactions available to the user. For example, the touch screeninput/display 103 may allow a user to move an input pointer or makeselections on the GUI by simply pointing at the GUI on the input/display103.

The GUI can be adapted to display a program application that requirestext input. For example, a chat or messaging application can bedisplayed on the input/display 103 through the GUI. For such anapplication, the input/display 103 can be used to display informationfor the user, for example, the messages the user is sending, and themessages he or she is receiving from the person in communication withthe user. The input/display 103 can also be used to show the text thatthe user is currently inputting in text field. The input/display 103 canalso include a virtual “send” button, activation of which causes themessages entered in text field to be sent.

The input/display 103 can be used to present to the user a virtualkeyboard 105 that can be used to enter the text that appears on theinput/display 103 and is ultimately sent to the person the user iscommunicating with. The virtual keyboard 105 may or may not be displayedon the input/display 103. In an embodiment, the system may use a textinput system that does not require a virtual keyboard 105 to bedisplayed.

If a virtual keyboard 105 is displayed, touching the touch screeninput/display 103 at a “virtual key” can cause the corresponding textcharacter to be generated in a text field of the input/display 103. Theuser can interact with the touch screen using a variety of touchobjects, including, for example, a finger, stylus, pen, pencil, etc.Additionally, in some embodiments, multiple touch objects can be usedsimultaneously.

Because of space limitations, the virtual keys may be substantiallysmaller than keys on a conventional computer keyboard. To assist theuser, the system may emit feedback signals that can indicate to the userwhat key is being pressed. For example, the system may emit an audiosignal for each letter that is input. Additionally, not all charactersfound on a conventional keyboard may be present or displayed on thevirtual keyboard. Such special characters can be input by invoking analternative virtual keyboard. In an embodiment, the system may havemultiple virtual keyboards that a user can switch between based upontouch screen inputs. For example, a virtual key on the touch screen canbe used to invoke an alternative keyboard including numbers andpunctuation characters not present on the main virtual keyboard.Additional virtual keys for various functions may be provided. Forexample, a virtual shift key, a virtual space bar, a virtual carriagereturn or enter key, and a virtual backspace key are provided inembodiments of the disclosed virtual keyboard.

FIG. 2 illustrates a block diagram of an embodiment of the devicecapable of implementing the current invention. The device 100 maycomprise: a touch-sensitive input controller 111, a processor 113, adatabase 114, a visual output controller 115, a visual display 117, anaudio output controller 119, an audio output 121, sensor inputcontroller(s) 118, an audio input controller 120 and a camera inputcontroller 122. The sensor input controller(s) 118 can include 3-axis(X, Y and Z) accelerometers and gyroscopes. In other embodiments thedevice 100 may include a range of other controllers and other componentsthat may perform a wide number of functions.

Basic Input

In an embodiment of the current invention, the user will use an inputdevice which can be a touch screen to enter text. The system will assumea virtual keyboard, which may or may not be visible to the user. Thiswill have a map of different “virtual keys” and may resemble the layoutof a real keyboard, using QWERTY or some other keyboard layout likeDVORAK. The user will be able to input text by applying tap gestures onthe different virtual keys. The device will detect the locations of theuser's taps or the relative locations of multiple taps and produce typedcharacters on the screen. The user may tap on the input device one ormore times with each tap usually representing one key stroke. Thevirtual keyboard may or may not be visible on a display or screen. Inaddition to taps to input text, the user can also input gestures thatcan cause the system to perform specific functions. The touch screeninput can be fairly large in devices such as tablet computers. However,in other embodiments such as smart watches, the touch screen can be verysmall.

In an embodiment, the inventive system can perform multiple functionsusing the same detected gesture based upon the context of the gesture.For example, a single finger right swipe can be interpreted as one ofseveral functions including: space, autocorrect, next wordprediction/suggestion, punctuate, repeat punctuation and move cursor atend of a word.

With reference to FIG. 3, if the context of the right swipe is after aknown word that is in an appropriate location in a string of the inputtext, the inventive system may interpret the right swipe as a request toinput a space after the completed word. For example, the user can usethe inventive system and tap at points (1) 121, (2) 122 and (3) 123which are respectively near letters C, A and N on the virtual keyboard105. The system may initially display the exact input text “Can” 160corresponding the locations and sequence of the tap gestures on thedisplay 103. The system may automatically respond to this input byaltering the input text. Because this is the first word of a possiblesentence, the first letter “C” may automatically be capitalized. Thesystem may also automatically display possible intended words 151including: Can, Cab, Van, Ban, Fan, Fat, Dan and Cam on a possible wordarea 127 of the display 103. In this example, the text “Can” is acomplete and known word. Thus, the system can interpret a subsequentright swipe 131 gesture as a command to complete the word input and adda space after the word Can 160. The cursor 162 is shown a space to theright of the word Can 160 and the system is ready to input the nextword.

Basic Text Correction

In an embodiment, the system can alternatively interpret the right swipegesture as a request for auto-correction and a space. With reference toFIG. 4, the user can tap at points (1) 121, (2) 122 and (3) 123 whichare respectively near letters C, A and E on the virtual keyboard 105.The system may not provide any suggestions (as shown on FIG. 4), beforea right swipe gesture is input. The system may initially display theexact input text “Cae” 161 on the display 103 corresponding thelocations and sequence of the tap gestures on the virtual keyboard 105.

With reference to FIG. 5, the user may have entered the text describedwith reference to FIG. 4 and then made a right swipe gesture 131.Because the text “Cae” is not a word, the system may respond to thisgesture by attempting to correct the input text. In this example, thesystem changes the text Cae to Car 163. The system may alsoautomatically display possible intended words 153 including: Cae, Car,Far, Bar, Fat, Bad, Bed, and Fee on a possible word area 127 of thedisplay 103. The current suggested word “Cae” may be indicated bybolding the text as shown or by any other indication method such ashighlighting, flashing the text, bold, contrasting color, etc. In thisexample, the text “Cae” is bold or otherwise indicated in the 151 on thedisplay. The system can continue to make additional suggestions asletters are added or deleted by the user through the input touch screen.

In this example, the input text “Cae” 161 shown in FIG. 4 and thesuggested word “Car” 163 shown in FIG. 5 may not be what the userintended to write. The user can input an up or down swipe gesture tomanually correct the text input. In an embodiment, up swipes can causethe displayed word to scroll backwards amongst the group of suggestedwords and down swipes can cause the displayed word to scroll forwards.Thus, an up swipe can cause the text, Cae to be displayed again as shownin FIG. 4 and a down swipe can cause the next suggested word Far 165 tobe displayed as shown in FIG. 6. Continued down swipe gestures can causeadditional suggested words 155 which can include: Car, Far, Bar, Fat . .. to be displayed one at a time for each down swipe gesture. Theselected suggested word is displayed in the text field and once thedesired word is in the text field, the user can continue to typeadditional characters which can automatically cause a space is addedafter the selected word and displayed word Far 165.

Advanced Text Correction

In an embodiment, the inventive system can indicate an active word thatmay not be the word that is currently being typed. For example, withreference to FIG. 7, the user has typed the partial sentence, “Hellobeaitodul|”. The user inputs a single finger right swipe and theinventive system attempts to auto-correct the input. In this example,the system does not recognize the text beaitodul 170, so this text ischanged to “beautiful” as shown in FIG. 8. In this case, the text“beautiful” 171 is both the active word and the word that the cursor 162is with. The inventive system, which handles text manipulation,considers the space that follows entered words, an attribute to that“textblock”, so the space between the word and the cursor does notseparate the word from the cursor. The suggested words 155 can include:beaitodul, beautiful, healthful, masterful, serotonin.

With reference to FIG. 9, the user has subsequently typed, “Hellobeautiful worl|”. The user is almost through typing the word “world” andthe cursor 162 is a vertical line adjacent to the text, “worl” 175. Theword that the cursor is within is underlined and the “active” wordbeautiful 174 is in italics for illustrative purposes. In the actualsystem, the active word may not be in italics. In this embodiment,up/down swipes will change the suggestion for the complete input activeword “beautiful” 174, which is the last corrected full word. In the wordsuggestion area 127, the suggested words 155 still include, “beaitodul,beautiful, healthful, masterful, serotonin . . . . ”

With reference to FIG. 10, after the user has finished typing, thesentence will appear as, “Hello beautiful world”. Note that the word“world” 175 is now both the active word and the word that the cursor 162is with and an up swipe or a down swipe in this context will cause thesystem to change the word “world” 175. In the word suggestion area 127,the suggested words 155 include, “world wield works work's worms fielddorms would . . . . ”

With reference to FIG. 11, once the sentence is complete, the user maywant to edit the word “beautiful” 174. The user can manually move thecursor 162 anywhere inside any of the displayed words as illustrated. Inan embodiment, the cursor 162 can be moved by touching the display 103at the desired cursor 162 location. In this example, the user has movedthe cursor 162 between the letters e and a in the word beautiful 174 andthe displayed text is, “Hello be|autiful world.” Since the cursor 162has been manually moved to the previous word, the active word and theword associated with the cursor 162 is now “beautiful” 174.

With reference to FIG. 12, the up swipe gesture can change the word“beautiful” 174 to the previous suggestion, which, in this case andsince the system made the proper correction on the first try, the textthat was actually typed, “beaitodul” 176 and the sentence is: “Hellobeaitodul|world.” The system can then move on to add additional text.

Dictionary Functions

In an embodiment, the inventive system can be used to add and/or removewords from a system dictionary. With reference to FIGS. 13A-13B, a userhas written the sentence, “Adding qwerty” and now wishes to add qwerty177 to the dictionary. The exact user text input may always be the firstitem on the word correction suggestion list.

Since the word, “qwerty” 177 is currently at the first word of thesystem suggestion list, this text can be added to the diction byperforming an up swipe in this context. With reference to FIG. 13A, theword “qwerty” 177 can be added to the dictionary. The system may providea signal such as a display or an audio signal that the word has beenadded. In this example, the system displays the text, “Learned qwerty”178 in the suggested word area 127. In some embodiments, though notshown in FIG. 13A, the system displays the text “Learned qwerty” 178 inthe main text area. With reference to FIG. 13B, if another up swipe isperformed again, the word qwerty can be removed from the dictionary andthe system can display the text, “Forgot qwerty” 179 to confirm thedeletion. In this example, the “Forgot qwerty” 179 message is displayedin the main text area. In some embodiments, though not shown in FIG.13B, the system displays the text “Forgot qwerty” 179 in the suggestedword area 127. Swiping up or down, can also move the cursor 162 to theend of the word qwerty 177.

Punctuation Marks

In addition to text, the inventive system can be used to inputpunctuation marks. As discussed, after a word has been input, the usercan right swipe to add a space after the word. With reference to FIG.14, the system may also interpret a subsequent right swipe gesture 131(two right swipes 131) as a request to input a punctuation mark. Forexample, a right swipe 131 after a word has been input can cause aperiod “.” 195 to be input and displayed after the last displayed word.The system can also add a space after the period. However, subsequentsingle finger right swipes 131 or a single finger right swipe 131 andhold can remove the space and cause additional periods 195 to be inputand displayed as shown in FIG. 14. The next four right swipes 192 causefour periods “. . . .” 195 to be input.

With reference to FIG. 15, in other embodiments, the system may use adifferent gesture to cause a punctuation mark to be added after the lastinput word “saw.” For example, in an embodiment, single finger rightswipes can cause the described word spelling correction and a completedword followed by a two finger right swipe 281 can cause the system toinput a period “.” 195 punctuation mark followed by a space.

In the suggested word section 127, a plurality of alternativepunctuation marks 156 can be displayed such as: . , ? ! 's : ; . Byinputting up and/or down swipe gestures as described above, thedifferent punctuation marks can be selected and input. In theillustrated example, the user has typed, “I don't know what I just saw”.The user then one finger or two finger right swiped 281 after the lastword was input to indicate a punctuation mark. The user may wish tochange the last period to a comma 196. Inputting a single down swipecauses the last period “.” 195 to be changed to a comma “,” 196 asillustrated in FIG. 14.

In general, a period is the normal input in response to two right swipegestures. However, in other embodiments, the system may intelligentlyinput a predicted punctuation mark based upon the context of the text.With reference to FIG. 16, the user has input the text, “Can you send methe document” followed by a single finger right swipe gesture 191 or atwo finger right swipe gesture 281. The system can interpret the inputtext “Can you send me the document” as a question and respond to thesingle finger right swipe gesture 191 or two finger right swipe gesture281 by inputting a question mark “?” 197 followed by a space.

In some embodiments the system can analyse the input words and determinethat a question mark is the most appropriate punctuation mark. Forexample, sentences that start or include typical “interrogative words”such as: Which, How, What, Why Where, When, Will, Are, Can, Is, etc. canindicate that the sentence is a question. In an embodiment, theinventive system can detect the presence of interrogative words and mayinput a question mark “?” rather than a period “.” in response to apunctuation mark gesture. If this is not what the user intended, thesystem can change the punctuation mark to any of the alternativepunctuation marks 156 . , ? ! 's : ; with single finger up and/or downswipes as described above.

In another example shown in FIG. 17, the system may interpret text inputin all capital letters and/or words that would be appropriate for usewith an exclamation point. In this example, the input text is, “WOW THATIS AWESOME” followed by a single finger right swipe gesture 131 or a twofinger right swipe gesture 281. The system can interpret this text “WOWTHAT IS AWESOME” context and determine that the most appropriatepunctuation mark is an exclamation mark “!” 198 followed by a space. Inyet another embodiment, the system can detect repeating the right swipegestures 191, 281 and display repeated punctuation marks. In thisexample, the system can detect multiple right swipes 191, 281 after acomplete sentence is detected, the system can display repeatedexclamation marks “!” 198. The displayed text can be “WOW THAT ISAWESOME!!!!” in response to the described text input and a right swipe191, 281 followed by four repeated right swipes 191, 281. In anotherembodiment, a right swipe and hold gesture can cause the system toremove the space and repeating the input of the prior punctuation mark.

Repeat Input Gestures

In an embodiment, the user can alter the right swipe gesture by holdingthe end of the swipe rather than releasing after the swipe. For example,the right swipe and hold can include touching the touch screen andsliding the finger across the surface and holding the finger against thesurface at the end of the right swipe. As discussed above, the rightswipe can cause various commands to be detected by the system. Therepeated action can be based upon the last right swipe input. Forexample, if the last input was a space, the swipe and hold gesture canresult in inputting multiple spaces. Alternatively, if the last inputwas a period, “.” the swipe and hold gesture can result in inputtingmultiple periods. The rate of repetition of the input can vary with theduration of the hold gesture. For instance, the input rate of spaces orperiods as described could happen faster the longer the user continuesthe hold gesture.

Deletions

In an embodiment, the present invention allows the user to actuate adelete function through an input gesture. The delete function can beeither a letter or the entire word based upon the context of the inputtext and/or position of the cursor relative to the text as described inFIGS. 17-20. In order to distinguish the deletion of a letter or a word,the system may only perform the letter delete function when the user hasperformed a left swipe while in the middle of inputting a word. When theword is not complete and/or not recognized as a full word by the system,each left swipe may have the effect of removing a single text character.

In the example shown in FIG. 18, the user has typed the text, “Yey” 164.The cursor 162 is adjacent to the last y. The user has then input asingle finger left swipe 132 which removed the last y and changed thetext to “Ye” 167 as shown in FIG. 19. However with reference to FIGS. 20and 20, when the left swipe is performed after a complete word has beeninput, the system can delete the whole of that preceding word. FIG. 20shows a device 100 with the word, “Year” 168 input and a space is placedbetween the word “Year” 168 and cursor 162. The user performs a leftswipe 132 and the entire word “Year” is deleted as shown in FIG. 21.Thus, the system can automatically determine if the single letterbackspace or full word delete function should be applied to the leftswipe gesture 132 based upon the context of the input.

While the user is typing a word, he or she may tap and input anincorrect letter. Because the user is in the process of typing the word,the cursor can be located directly adjacent to the last input letter.The user can notice this error and input a left swipe gesture 132. Thesystem can detect the gesture and the position of the cursor next to aletter. Based upon this context, the system can remove a single letterto the left of the cursor. If additional left swipe gestures aredetected, an additional letter for each swipe can be deleted. Aftermaking the correction described above with reference to FIGS. 18 and 19,the user may then tap on points (3) 181 and (4) 184 corresponding to theletters “a” and “r” respectively as shown in FIG. 20. The output ismanually corrected to add onto points “Y” (1) 122 and “e” (2) 125,followed by (3) 181 and (4) 184 corresponding to letters “a” and “r” andresulting in the text “Year” 168 shown in the display 103.

In some embodiments, the system may enable a “continuous delete”function. The user may invoke this by performing a combination gestureof a left swipe and a hold gesture at the end of the left swipe. Thefunction will have the effect of the left swipe, performed repeatedlywhile the user continues holding his finger on the screen at the end ofthe left swipe (i.e. while the swipe and hold gesture is continuing).The repetition of deletions could vary with the duration of the gesture.For instance, deletions could happen faster the longer the user has beencontinuing the gesture. If the delete command is a single letter deletebackspace, the deletion may start with single character by characterdeletions and then progressing to delete whole words after apredetermined number of full words have been deleted, for example one tofive words. In the single letter and full word delete modes, each of theletters and words may be deleted at a predetermine rate of time betweeneach letter or word deletions. However, as more letter or words aredeleted, the system can increase the speed which the letters or wordsare deleted.

Word Prediction

In an embodiment, the inventive system can use at least some of thedescribed gestures to perform “word prediction” processing. This featurecan be particularly useful with smart watch embodiments where individualletter input can be more difficult than devices having larger touchscreens. With reference to FIG. 22, the user has typed the input text“hsw”. The system can respond to this input text by recognizing thatthese letters are not a word and suggesting a set of possible correctedwords “Hsw How Joe His For Now The Bow” listed in the suggested wordsection 127. The first suggested word “How” can be in bold to indicatethat it is the current selected recommended replacement word.

In an embodiment, with reference to FIG. 23, the user can then perform asingle finger right swipe gesture 131 to choose the first recommendedreplacement word “How” and inputting a space after the input replacementword. The system has input a new set of suggested predicted words in thesuggested word section 127 which in this embodiment include, “are didmany is can should much to.” In an embodiment, the inventive system canperform the illustrated subsequent word prediction through an n-gramprediction process. In a simplified embodiment, the inventive system canhave a database which stores a bigram which provides a probability orfrequency of two word combinations. A portion of the bigram isrepresented by Table 1 below. In FIG. 23, the predicted words can bedisplayed in descending order of probability that the word combinationis the predicted word to be input. The bigram for the first word How canextend beyond the eight predicted words in Table 1 and these additionalpredicted words can be displayed in the suggested word area 127 byperforming the up and/or down swipes described above.

TABLE 1 Word Combination How How How How How How How How are did many iscan should much to Frequency 1,000 741 573 447 303 278 193 54

In other embodiments, the inventive system can utilize one or more of:bigram, trigram, quadgram or n-gram word prediction systems that woulduse tables having frequencies or probabilities for word combinations forthree or more word text sequences. The n-gram word predictions can use auniversal database for all users or alternatively, the system can usedatabases that are customized for one or more specific users based uponthe statistical analysis of the users' prior writings. In otherembodiments, any other suitable word prediction system can be used bythe system.

With reference to FIG. 24, the user has reviewed the suggested words andagreed with the system's first predicted word “are” by inputting asingle finger right swipe 131. The system inputs the word “are” followedby a space. The system has also displayed a next set of predicted words,“you the these our their your my” in the suggested word section 127.Again, the displayed predicted words can be based upon an n-gramprediction process such as a bigram or a trigram an example of which isbelow in Table 3. With reference to FIG. 25, the user may disagree withthe first predicted word “you” but see the desired word in the suggestedword section 127. The user may input two down swipes 133 to change theselected word to “these” which is added to the displayed word sequence.The user can finalize the selection of the suggested word “these” with aright swipe gesture 131 which also causes a space to be input after thelast input word.

TABLE 2 Word Combination How How How How How How How are are are are areare are you the these our their your my Frequency 1,320 771 637 619 428403 92

With reference to FIG. 26, the user may wish to finish this sentence andcan input a two finger right swipe 281 which can cause a period “.” tobe input. However, the user may want a question mark instead. Withreference to FIG. 27, the user can input two single finger down swipes133 which can cause the selected punctuation mark to move two to theright from “.” to “?” in the selected punctuation mark section 127. Theuser can finalize this selected “?” punctuation mark with a right swipe131 or by inputting text letter taps. As discussed above, in someembodiments the system can analyse the input words and determine that aquestion mark is the most appropriate punctuation mark. For example,sentences that start or include typical “interrogative words” such as:Which, How, What, Why Where, When, Will, Are, Can, Is, etc. can indicatethat the sentence is a question. In an embodiment, the inventive systemcan detect the presence of interrogative words and may automaticallyinput a question mark “?” rather than a period “.” in response to apunctuation mark gesture. Although the punctuation mark input has beendescribed as a single or two finger right swipe gesture, in otherembodiments the inventive system can input the punctuation mark inresponse to any other suitable detected gesture input.

In yet another embodiment, the question mark may be part of thesuggested “words” based upon an n-gram word prediction. For example,with reference to FIG. 28, the system can display a mixed group ofpredicted words and punctuation marks in the suggested word area 127.For example, in an embodiment, the system may respond to the text input,“How are these” by displaying a set of next suggested words that includeboth words and at least one punctuation mark. In this example, thesystem displays, “related ? used items opinions things”. The user canthen input a single finger down swipe 196 to switch the suggested wordfrom “related” to “?” in the suggested word area 127 and the input text.The user can then input a right swipe 131 to input the “?” followed by aspace. The system can repeat the described process for the nextsentence.

In this embodiment, the system may use a bigram, trigram or quadgramanalysis as shown in Table 3 that includes both words and the questionmark punctuation mark.

TABLE 3 Word Combination How How How How How are How are are are arethese are these these these these related these? used items opinionsthings Frequency 653 551 498 451 308 9

The n-gram can be based upon a cumulative writings of many authors anddocuments. In other embodiments, the n-gram analysis can be based upon asmaller group of documents from a limited number of writers. In yetanother embodiment, the system may produce the n-gram analysis basedupon the prior writings of the author.

Keyboard Functions

In some embodiments, gestures may invoke different keyboards in arepeating rotation. For example, system may include multiple keyboardswhich can be changed by the user. The “normal” letter character keyboardmay be the default keyboard. The normal keyboard can be changed to anumeric keyboard, which may in turn be changed to a symbol keyboard. Thesystem may include any number of additional keyboards. In an embodiment,the inventive system may include dedicated keyboard buttons that can beused to change the keyboard. The keyboard switching cycle can berepeated as necessary. In an embodiment, the user can configure thesystem to include any type of keyboards. For example, there are manykeyboards for different typing languages.

In different embodiments, the system can recognize certain gestures forquickly changing the layout of the keyboard and/or invoking settingsmenus. For example, a two finger up/down swipes may be used to scrollthrough alternative keyboards layouts. With reference to FIG. 29, anembodiment of a full keyboard 106 is illustrated that includes allletters in a QWERTY layout and below the keyboard are buttons for: shift108, keyboard toggle 141, space 140, backspace 144 and new line 142.This keyboard layout can also include icons for UI notification.

In an embodiment of the inventive system, the user can scroll throughthe keyboard layouts with two finger up/down swipes on the touch screen103, the system can display alternative keyboard layouts. With referenceto FIG. 30, an alternative keyboard 105 layout with the first row ofkeys below bottom letter row removed. In this example, the shift key 108has been moved next to the “Z” key. With reference to FIG. 31, the nextalternative keyboard can be an invisible keyboard. In this example theletters, shift, keyboard, space, backspace and new line have all beenremoved. The only visual graphics include the word suggestion bar andsome UI notifications and animations which can indicate controls such asshift. With reference to FIG. 32, the next keyboard is even more minimalwith the word suggestion bar removed and just some animations and UInotifications still visible. In this example, the shift key UInotification 107 is shown which indicates that the shift key has beenactuated and the next input letter will be a capital letter. The systemcan scroll through all of the possible keyboards in a loop and back tothe initial keyboard as shown in FIG. 29 with repeated two fingerup/down swipes.

As discussed above, the system can include a “keyboard toggle.” In anembodiment, pressing the keyboard toggle causes the keyboard to switchor scroll through a series of alternative keyboards. With reference toFIG. 33, a second keyboard can be a first symbols keyboard 222. Withreference to FIG. 34, a third keyboard can be displayed which can be anadvanced symbols keyboard 223. The alternative keyboards can include a“left arrow keyboard” button 220 and a “right arrow keyboard” button221. The user can click the left arrow keyboard 220 to go the priorkeyboard and the right arrow keyboard button 221 to go to the nextkeyboard.

In other embodiments, the type of the swipe may control the way that thekeyboard is changed by the system. For example, from a text keyboard,two finger right/left swipes may invoke different language keyboards.Similarly, from a symbols keyboard, two finger right/left swipes mayinvoke different symbol keyboards. Again, these multiple keyboards canbe configured in a loop so that each of the keyboards can be accessedthrough repeated two finger swipes in one direction.

Many mobile devices are small enough to be held with one hand. Thus, theuser will typically hold the device and input text with a thumb. If themobile device is large and the user's hand is small, it can be difficultfor a right thumb to touch the left side of the mobile device or a leftthumb to accurately touch a right side of the mobile device. In anembodiment, the inventive system can be used to alter the left/rightposition of the keyboard on the touch pad of the mobile device to aposition and scale that is most comfortable to the user. The system maybe in a normal text input mode with the keyboard aligned with the middleof the display. The user can input a two finger right swipe 281 and thesystem can respond by moving compressing the keyboard to the right sideof the display. The system can display the compression of the keyboardduring the swipe so the user can easily control the level of compressioncan be based upon the length of the swipe. With reference to FIG. 35, atwo finger right swipe 281 can cause the keyboard to compress againstthe right side of the display 103. In an alternative embodiment, a threefinger right swipe may be used to cause the keyboard to compress againstthe right side of the display 103. This can allow for easier and morecomfortable typing for right handed users. With reference to FIG. 36, atwo finger left swipe 282 or alternatively a three finger left swipe 284can cause the keyboard to compress against the left side of the display103.

Smart Watch Applications

In an embodiment, the inventive system described above can be used withdevices having very small touch screen inputs. FIGS. 37-38 illustrateembodiments of the present invention used with a smart watch 200 thatincludes a touch screen 103 input device. FIG. 37 illustrates anembodiment of the touch screen 103 that is very small with allunnecessary feature buttons minimized or hidden. In this configuration,the smart watch 200 may only display a very small and basic QWERTYkeyboard 105. The user can one finger type on the keyboard 105 in themanner described above and the system will detect the input letters andgestures. The functionality will be very similar to the normal systemdescribed above but gestures may be more important because specialfunction keys may be omitted from the small display 105.

In this example, the user has typed or input by other means the text, “Imake typing”. The system can display a set of suggested words including:typing, tycoons, typhoons and tykes. The user can then input a singlefinger right swipe 131 to accept the first suggested word “typing” andadd a space. For example, the system may display the text, “I maketyping” and lists a set of predicted words including: awesome, fun,enjoyable and efficient. In this example, the user can respond to thesepredicted words by inputting a single finger right swipe gesture toaccept the predicted word “awesome”. However, the user can switch thepredicted word with up/down swipes as described above. In anotherexample, the user can then input a two finger right swipe 281 which cancause the system to display a set of punctuation marks and display thefirst predicted punctuation mark at the end of the text, “I make typingawesome.” The user can switch the punctuation mark to any othersuggested punctuation mark with up/down swipes as described above.

The left swipe functionality in the smart watch embodiment, can besubstantially the same as described above. When the cursor 162 is at theend of a word, a left swipe can delete the entire word. However, if thecursor 162 is in the middle of the word, a left swipe may only delete asingle character or letter for each left swipe. It is possible to repeatthis function by performing a left swipe and hold so that a plurality ofcharacters or words can be deleted with a single left swipe and holdgesture.

In yet another embodiment, the smart watch can indicate a plurality ofsuggested word corrections or predicted words through a voice output.The user can then scroll through the suggested word corrections byperforming up and/or down swipes on the watch face as described abovewhich can prompt the system to output an audio indication of the currentsuggested or predicted word. Down swipes can cause the next suggested orpredicted word to be indicated and up swipes can cause the previoussuggested or predicted word to be indicated. The up/down swipes can alsobe used to add or remove words from the system dictionary in the samemanner and context as described above with reference to FIGS. 13A and13B.

Because smart watches can have very small screens, some normal buttonfunctions can be changed to gesture functions in the smart watchembodiment. For example, a normal virtual keyboard may have a dedicatedshift key to control upper and lower case letter inputs. The smart watchembodiment may remove the shift key and interpret a two finger up swipeas the shift key toggle. If the user needs to lock the shift key, thecorresponding control gesture can be two finger up swipe and hold. A twofinger swipe down gesture can hide or retract the keyboard. The shiftlock can be dismissed with a swipe up with two fingers. In anembodiment, the screen may display an icon indicating the shift lockengagement.

The inventive system can utilize a displayed virtual keyboard shown inFIG. 29 or an invisible keyboard shown in FIG. 31. The smart watch mayalso include other variations similar to the keyboards described abovewith reference to FIGS. 30-34. In an embodiment, the user can alter thedisplayed keyboard between a completely invisible to a maximum sizekeyboard that includes the spacebar row by performing swipe up/down andhold gestures. Swiping up and holding can cause scrolling in a firstdirection through the available keyboards and swiping down and holdingcan cause scrolling in an opposite direction.

As discussed above, an embodiment of the present invention uses akeyboard button to shift keyboards. In the smart watch embodiment, twofinger taps may cause the system to scroll through different types ofkeyboards which can be similar to the keyboards described above withreference to FIGS. 29-34. There can be various keyboards includingletters, numbers and symbols. If the user passes a desired keyboard, heor she can get back to this keyboard by repeating the two finger tapgesture. A two finger tap and hold can be used by the system to enable avoice to text function. Repeating the two finger tap and hold can causethe system to disable the voice to text function.

If the user taps and holds on a specific letter, number, symbol orcharacter on a keyboard, the system can respond by displaying variationson the selected text. For example, if a user taps and holds on the “a”key, the system may display a set of similar letters such as: á, à, â,ä, ã, and å. The system may perform a different function if the usertaps and holds in a different context. For example, if the user taps andholds on input text, the system can respond by magnifying the selectedtext away from the finger so that the user can check for accurate input.In addition to taping and holding, the inventive system may perform thedescribed functions by tapping and dragging the finger to the area ofinterest on the smart watch display.

The smart watch may be able to control various different programapplications that perform various different functions. Thus, it would beuseful for the described gestures to provide different functionality indifferent applications. For example, as shown in FIG. 38, in text inputapplications such as email, text processing, etc., a right swipe andhold gesture can cause a new line to be input. This can be similar to areturn button on a normal keyboard. However, in a search applicationsuch as Google, the right swipe and hold gesture can indicate that theyuser wants the program to perform a search. In other applications, theright swipe can cause a program or a portion of a program to go or run.

In other embodiments, the system can add these sensor inputs to thesystem's recognized controls. As discussed, the inventive smart watchembodiment, may also utilize other sensors such as accelerometers orgyroscopes that can be used as input control signals in addition to thetaps and gestures on the touch screen as discussed above with referenceto FIG. 2. The internal accelerometers may detect taps on the watch orother movements of the watch and the gyroscopes can detect rotation ofthe watch. For example, the system may perform word correction and spacefunction in response to an accelerometer detected taps anywhere on thewatch. Thus, a user would input the letters for a word and then tap onany non-keyboard portion of the smartwatch to cause the system to inputa space after the word. This function can be substantially the same asthe functions described above with reference to the right swipe gesture.

Other sensor detected user and watch movements can cause the smart watchto perform other functions. For example, the X and Y axis gyroscopes candetect rotation about the X and Y axis of the smart watch. The user'sarm can be substantially aligned with the X axis and perpendicular tothe Y axis. Thus, a left hand downward wrist rotation movement can bedetected as a counter clockwise rotation about the Y axis and an upwardwrist rotation movement can be detected as a clockwise rotation aboutthe Y axis by the Y axis gyroscope. Similarly, the outward rotation ofthe hand can be detected as a clockwise rotation about the X axis andinward rotation can be detected as clockwise rotation about the X axisby the X axis gyroscope. These rotations can correspond to the differentdirection single finger swipe functions described above. Morespecifically, the right swipe functions can be performed in response toa left wrist down rotation, left swipe functions can be performed inresponse to the left wrist up rotation, up swipe functions can beperformed in response to outward rotation and down swipe functions canbe performed in response to inward rotation.

More specifically, the smart watch can perform the described wordcorrection and space input functions in response to the left wrist downmovement and perform letter or word delete functions in response to leftwrist up movements. In the suggested word mode, outward and inward wristtwists can allow a user to scroll through suggested words, as describedabove with the up/down swipes. The described swipe and hold functionscan also be performed through physical movement of the smart watch. Forexample, moving and holding the wrist can cause the system to performthe corresponding described swipe and hold functions.

In an embodiment, the system may recognize that the described text inputmethods can only function when the smart watch is in a limited range ofpositions typically with the face in an upward position. The system maygo into a sleep mode or other energy saving modes when the watch is notin an operating position or if rapid motion is detected indicating thatthe user is currently performing an athletic task.

Accessibility Modes

The system may use a device other than a screen to provide the feedbackto the user. For instance, the present invention may be employed with anaudio output device such as speakers or headphones. In certainembodiments of the invention, the user will type using the usual tapgestures. The device may provide audible signals for each tap gesture.Once a rightwards swipe is given by the user, the system will correctthe input and read back the correction using audio output. The user maythen apply the upwards/downwards swipe gestures to change the correctionwith the next or previous suggestion, also to be read via audio outputafter each gesture. Such an embodiment may allow use of the invention byvisually impaired user, or may enable its application in devices withoutscreens, or by users who prefer to type without looking at the screen.

In an embodiment, the inventive system may include an audio output andmay also provide audio feedback for some or all of the additionalfunctions described above. For instance, the deletion of words asdescribed could be announced with a special sound or voice, and deletionof characters could be indicated with a different sound or voice. Manymobile devices such as cell phones also have a vibration feature thatcan be used by the inventive system to provide motion feedback when textinput functions are actuated. In other embodiments, a variety of soundsand/or vibrating feedback could be used in response to different swipinggestures input by the user and detected by the inventive system.

Because the visually impaired may not be able to see the keyboardsdisplayed on the screen, the inventive system may include gesturecontrols for inputting common and necessary text commands. For example,the system may interpret a two finger down swipe as a new line input.The system may also interpret a two finger up swipe as a signal toinvoke a menu. The system may then announce the menu rather thandisplaying the menu so the user can respond to the audio prompts bytapping on the touch screen to select the desired menu item. In anembodiment, the inventive system may detect the user of special devicefeatures such as Voice Over on iOS or Talkback on Android and activatethe accessibility features based upon the user's actuation of theseaccessibility features.

Three Dimensional Space Gestures

While the inventive system has been described with a touch screen input,in other embodiments, the inventive system can be implemented throughother input devices such as cameras which may be built into the device.With reference to FIG. 39, in an embodiment body movement or fingergestures of a user can be obtained using an optical device comprising animage camera 551, an infrared (IR) camera 553 and an infrared (IR) lightsource 555 coupled to a signal processor. The IR light source 555, IRcamera 553 and an image camera 551 can all be mounted on one side of theoptical device 550 so that the image camera 551 and IR camera 553 havesubstantially the same field of view and the IR light source 551projects light within this same field of view. The IR light source 555,IR camera 553 and image camera 551 can be mounted at fixed and knowndistances from each other on the optical device 550. The image camera551 can provide information for the patient's limb 560 or portion of thepatient within the viewing region of the camera 551. The IR camera 553and IR light source 555 can provide distance information for each areaof the patient's limb or digits 560 exposed to the IR light source 555that is within the viewing region of the IR camera 553. The infraredlight source 555 can include an infrared laser diode and a diffuser. Thelaser diode can direct an infrared light beam at the diffuser causing apseudo random speckle or structured light pattern to be projected ontothe user's body 560. The diffuser can be a diffraction grating which canbe a computer-generated hologram (CGH) with a specific periodicstructure. The IR camera 553 sensor can be a CMOS detector with aband-pass filter centered at the IR laser wavelength. In an embodiment,the image camera 551 can also detect the IR light projected onto theuser's limbs, hands or digits 560.

In an embodiment the system may include a user interface that allows auser to configure the inventive system to the desired operation. Thedescribed functions can be listed on a settings user interface and eachfunction may be turned on or off by the user. This can allow the user tocustomize the system to optimize inputs through the touch screen of theelectronic device.

It will be understood that the inventive system has been described withreference to particular embodiments, however additions, deletions andchanges could be made to these embodiments without departing from thescope of the inventive system. Although the order filling apparatus andmethod have been described include various components, it is wellunderstood that these components and the described configuration can bemodified and rearranged in various other configurations.

What is claimed is:
 1. A method for inputting text, comprising: at a computer system having one or more processors, a touch screen, and memory storing programs for execution by the one or more processors, displaying a virtual keyboard on the touch screen; detecting user touches on the virtual keyboard; in response to the user touches, generating text input to the computer system and displaying the text input as it is generated; detecting a swipe gesture on the touch interface, the swipe gesture including an initial touchdown point and a direction; determining the direction of the swipe gesture; determining a context of the swipe gesture based upon the text input, wherein the context includes one of the text input representing a partial word, a complete word, and a sentence; determining a type of the swipe gesture, wherein the type includes one of a single finger swipe gesture, a two finger swipe gesture, and a three finger swipe gesture; and performing a predetermined editing function in relation to the text input based on a respective combination the determined direction, context and type of the swipe gesture.
 2. The method of claim 1, further comprising: displaying a suggested word on the touch screen based on the text input; and when the context is the text input representing a partial word, in response to a first distinct combination of the determined direction and type of the swipe gesture; changing the text input to the suggested word and displaying the suggested word on the touch screen.
 3. The method of claim 1, further comprising: when the context is the text input representing a complete word, in response to a second distinct combination of the determined direction and type of the swipe gesture, displaying a first predicted word on the touch screen after the displayed complete word.
 4. The method of claim 1, further comprising: when the context is the text input representing a partial word, in response to a third distinct combination of the determined direction and type of the swipe gesture; deleting a last character of the text input and displaying the text input without the deleted last character.
 5. The method of claim 3, further comprising: when the context is displaying the first predicted word, in response to a fourth distinct combination of the determined direction and type of the swipe gesture, ceasing display of the predicted word on the touch screen.
 6. The method of claim 3, further comprising: when the context is displaying the first predicted word, in response to a fifth distinct combination of the determined direction and type of the swipe gesture, displaying a second predicted word on the touch screen instead of the first predicted word.
 7. The method of claim 6, further comprising: when the context is displaying the second predicted word, in response to the fifth distinct combination of the determined direction and type of the swipe gesture, displaying a third predicted word on the touch screen instead of the second predicted word.
 8. The method of claim 3, further comprising: selecting for display the first, second and third predicted words such that the first predicted word has a higher n-gram probability than the second predicted word and the second predicted word has a higher n-gram probability than the third predicted word.
 9. The method of claim 1, wherein the first predicted word and a plurality of predicted words are determined by a highest n-gram probability based on the input-text.
 10. The method of claim 9, wherein the n-gram is a bigram, a trigram or a quadgram.
 11. The method of claim 1, further comprising: when the context is displaying the sentence, displaying a punctuation mark is included in the first predicted word or the plurality of predicted words.
 12. The method of claim 1, further comprising: in response to a sixth distinct combination of the determined direction and type of the swipe gesture, ceasing display of the virtual keyboard and replacing the virtual keyboard with a second keyboard layout that is invisible.
 13. The method of claim 12, wherein the virtual keyboard is a QWERTY keyboard and the second keyboard is not a QWERTY keyboard.
 14. The method claim 1, wherein the computer system is a mobile device, the touch screen is very small, and the device includes at least one of an accelerometer and a gyroscope, further comprising: detecting with the accelerometer gestures that include taps on the device or motion of the device; and detecting with the gyroscope gestures that include rotations of the device.
 15. The method of claim 14, wherein the device includes a microphone, further comprising: in response to a seventh distinct combination of the determined direction and type of swipe gesture, actuating a voice to text input function and enabling a user to provide the text input using the microphone.
 16. The method of claim 14, wherein the device is a smart watch.
 17. An electronic device, comprising: one or more processors; a touch screen; and memory storing programs for execution by the one or more processors, the programs including instructions for: displaying a virtual keyboard on the touch screen; detecting user touches on the virtual keyboard; in response to the user touches, generating text input to the computer system and displaying the text input as it is generated; detecting a swipe gesture on the touch interface, the swipe gesture including an initial touchdown point and a direction; determining the direction of the swipe gesture; determining a context of the swipe gesture based upon the text input, wherein the context includes one of the text input representing a partial word, a complete word, and a sentence; determining a type of the swipe gesture, wherein the type includes one of a single finger swipe gesture, a two finger swipe gesture, and a three finger swipe gesture; and performing a predetermined editing function in relation to the text input based on a respective combination the determined direction, context and type of the swipe gesture.
 18. The device of claim 17, further comprising: in response to a distinct combination of the determined direction and type of the swipe gesture, ceasing display of the virtual keyboard and replacing the virtual keyboard with a second keyboard layout that is invisible.
 19. The device of claim 18, wherein the virtual keyboard is a QWERTY keyboard.
 20. The device of claim 17, wherein the device is a mobile device, the touch screen is very small, and the device includes at least one of an accelerometer and a gyroscope, further comprising: detecting with the accelerometer gestures that include taps on the device or motion of the device; and detecting with the gyroscope gestures that include rotations of the device.
 21. The device of claim 20, wherein the device includes a microphone, further comprising: in response to a distinct combination of the determined direction and type of swipe gesture, actuating a voice to text input function and enabling a user to provide the text input using the microphone.
 22. The device of claim 20, wherein the device is a smart watch.
 23. A non-transitory computer readable storage medium storing one or more programs, the one or more programs comprising instructions, which when executed by an electronic device with a touch screen, cause the device to perform operations comprising: displaying a virtual keyboard on the touch screen; detecting user touches on the virtual keyboard; in response to the user touches, generating text input to the computer system and displaying the text input as it is generated; detecting a swipe gesture on the touch interface, the swipe gesture including an initial touchdown point and a direction; determining the direction of the swipe gesture; determining a context of the swipe gesture based upon the text input, wherein the context includes one of the text input representing a partial word, a complete word, and a sentence; determining a type of the swipe gesture, wherein the type includes one of a single finger swipe gesture, a two finger swipe gesture, and a three finger swipe gesture; and performing a predetermined editing function in relation to the text input based on a respective combination the determined direction, context and type of the swipe gesture. 